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ABSTRACT 

This interim report is concerned with the analysis of 
educational experiments and qaasiexperiients where alternative 
teaching aethods are applied to intact classes or where alternative 
prograss are set np in sasples of schools or coatunities* Generally, 
such studies have used the class, the school, or the coaaunity as the 
nnit of saapling with the consequence that investigators have based 
i;he nuaber of degrees of freedoa in their statistical calculations on 
the nuaber of classes. However, the author believes that a further 
distinction aust be aade: that is, between effects that operate at 
the class level and effects that operate within the class. The 
variance in an outcoae aeasure can be divided into between^class and 
within-class coaponents. The between-class variance can in turn be 
divided into variance predicted froa the aptitude level of the class 
and a residual. The within-class variance can be subdivided into the 
effect predicted froa the within-class regression equation and a 
residual. The regression equation with each single c^^ass can then be 
coaputed and the residual reduced still further. The author has 
developed aethods for coapleting such analysis and applies thea to 
previous studies to show the effect of such analysis on the outcoaes 
of studies. (HRD) 
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This project was under taken as a "tool ing-up" year, to define 
questions iv.are prociscly and to develop tecliniques that could be 
applied to the rcanalysis of laiT^c-Kcnlc educational studies. The work 
has piof^rcr.Hod as far as was expected during the year» although the 
spuciCic questions that cotUTinnded attention have shifted somewhat. A 
grant to continue tli:» work was approved by the Spencer "^ouiidation, and 
thi.s docni:iL?nt la an interim report even though for administrative pur- 
poses, the orifiLnal grant is concidcred to be terminated. 

Joseph G. Dckcn, a doctoral student in the Department of Statistics, 
played a inajor role in the work reportt^d. Koreen Webb, a doctoral 
Ktudent in educational psychology and measurement, joined the pi-oject 
in September and will continue through the next year. She helped in the 
preparation of this report. 
Oycrvjcv; 

This project Is concerned with the analysis of edu-atioual 
experiments and quasiexperiments where alternative teaching methods are 
applied to intact classes, or where alternative programs are set up 
in samples of schools or communities. In such a study the class (or 
school or community) is the "unit of sampling". Recognition of that 
fact by investigators typically bas liad just one consequence; some of them 
have ba?;ed the number of degrees of freedom in their statistical calcu- 
lations on the number of clafines. Beyond this, we argue, n conceptual 
dintlnctif'U muiit be made: bctv^oen effects that operate "at the clar.R 
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It'xfci" and eCfocts thnt opcrnto withi;. Jil- cla.^^s. This beconiCB critically 
important in eMaininiui\ thv ci'f oo.tr, of n treatment on different types of 
pupilfi. 

The varXunctJ in an outcor.iG measure (within a treatment) can be 
diviJud into betwcfn-claus and within-clnss components. The botv^een- 
cla.^s Vviriancc cnn in turn be divided inti? variar.ce predicted from the 
aptitude* level of the claa?? and a residual. Tn ;he process, the slope 
of thu outcome-on-aptiludc rcj^rcKsion between classes is examined. 
If the Hlopos in the two treatments differ. Aptitude x Treatment inter- 
action operates between classeB; then one might pursue a policy of 
assign Inr, high-aptitude classes to one treatment vhilv. applying the 
second trciiLmt'iit in low-aptitude classes. 

The within-class vnria'icc- (wlthju either treatment) can bo 
similarly subdivided into the ef/cct predicted from the pooled-within- 
class rer.ression equation, ana a ienJ dual . Thwi the regression equation 
withia each .single class caTi be computed, and the residual rcc'uced still 
further. The dependence of outcome on aptitude wir.hin the class nay 
reflect how individual ability responds to the treatment or it: may 
reflect social-psychological effects (e.g., response competition). 
Diffe.-ences in slope from one class to another (within a treatment) may 
be a ci'>nsequence of ascertainable differences in the teachers* practices. 

Xa the course of our research to date, we have identified some 
important decisions the data analyst needs to make in separating such com- 
ponents as the preceding paragraph describes. We have adopted one set of 
operations (r.rounded in SPSS computer programs), and have applied them to 
a si{;niric;}nt study by G. L. A'lderson, to demonsM;ate the kind of results 
expected. The new analysis undarir.incs Anderson's conclusion that hns 

*Aptitude is a formal term applying to any individual characteristic at 
pretest. In some studies demographic characteristics arc critical. 



stood L'ittco J9^U. This is imfoi ttmal huL it dcmousLrat cs the value of 
tho new 

Judj;»,mcn\t is required to rhoof^e betwe<Mi the altei-n;itives open 
to the data analyst. While the choicer; we made in the Anderson analysis 
are^ ve bclii?ve> appropriate for that study, ve Intend to spell out the 
racionaLe underlyiuy each choice, so that future investigators can make 
choice;, appropriate to their ow^n studies. For example, we have 
chosen to regard elastics as randomly sampled from a population^ and to 
regard students within classes as "fixed" in the statistical serje. 
Under somc» circumstances an investigator might prefer to rega**J both 
as random, or to regard classes as fixed and pupils as random. 

Our scheme leaves us in a position to test regression and inter- 
action effects for .significance. But ti?e have become increasingly con- 
vinced that statistical inference ought to emphasise confidence inter- 
vals rather than tents of the null hypothesis. 

We have studied PoLthoff^s (1954) extension of the Johnson- 
Neyman method. The method docs appear to be suitable, though with the 
data collected in typical educational studies the confidence limits for 
effect-^ turn out to be very wide. I.e., the typical investigation gives 
only sLimpy information on the strength of effects in the population. 
By way of demonstration, we have applied Potthoff ^s method to a study 
by Austin Bond. Since confidence intervals for regression effects are 
difficult to comprehend unless they are displayed visually, we are making 
use of the computer to plot confidence intervals with one or two pre- 
dictors. We have some further work to do to get displays that are 
adequately clear. 
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'iho uothod applies to Ihi? forij;it ion of confidence liitervali; for 
caclk wUliin~t real riont regrewBioii, as wt?] 1 as to the interactions 
Pottlioff conHidtTc'd. Uspt-^clally in quafsiaxperinicnto, this is likely to 
bt? uore. appropriate than to fo^u:* on the Inferoctiou directly. 

)Yora the point of view of the Foundalioii, the most important 
tiiatt.t^r to report is that vork has proceeded according to plan, that 
the tentative £indiiH'.fv have coi\firmed the importance of work along 
these linef>» and that wa expect highly useful results from the pucccssor 
grant, which v;lll extend our pilot work into a more, or less definitive 
repoft and will demotu-.tratt; the irvpor race of analysis hetwecn- and 
vifhin-clartrffcs. We shall elaborate in the follov/iuj; sections, primarily 
to r.J^ve collc: igues who sec the report n. more concrete idea of the direc- 
tion our work is taking* 

The work :ippeaj£: In have: iTiiplUat ioiin for a hind of question 
not considered in our propot^al. Few issues regarding methodology in 
educational research are more significant thaa the legitimacy of 
covarlance adjustmentn in quasiexpcriwents. Many of the controversies 
over, for example, the Westinghouse evaluation of Hoadstart had to do 
with this issue. The distinction between within-class and between- class 
regressions is not considered in the traditional analysis of covariance. 
We suspect that many analyses have inappropriately calculated regressions 
for purposes of adjustment by pooling individuals from all classes within 
treatments. While work on this extension of our thinking is not scheduled, 
we mention it here to indicate that our program of investigation has 
wider significance than our proposals have indicated. As ® ^na^^^er of fact, a 
current request for propo-.al's from the Office of Education, for the Follow 
Through i'ianned Variation quasiexperiment, asks explieitly foi analysis- of both 
main effects and interactions at the between and within site levels. So our 
ERJC results will be ready none too soon! 
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Cronbach ami J»uow had worked iroixt 1.966 on studicB on individual 
differeiii!w>s in rospoune Lo iiuaruction* Their mouograpli ApiitudcH find 
Instructi o nal Hothods is uenrit^g completion and will pe sent to press 
(Irvii\f,t:on rrcstr^, New York) within a fuw weeks* It has been possible 
to insert into the manuscript a brief account of the work on this 
project > includinj; specifically the findings and mathematical foinnula- 
tlons suminarised below. Since the monograph is expected to influence 
subsequent inve.^?tigntioas, this grant has served to increase signifi- 
cantly the nathodolo^lcal soundness and the probable benefit from the 
Cronbach*«Snuv; prograu of work, 

A paper by Cronbach should also be mentioned* His invited 
addreas to the Amcricau Psycholo;;xcal Association in September^ 1974 
will be pabl:!5,bc^d in the Fehruuy, 1975, American Psychol ogist under the 
title ••Beyond the two disciplnno^: of scientific psychology". That paper 
does not derive directly from the grant, yet it addresses the large 
issue to which the research is relevant: Wiat are the limits of inter- 
pretation of educational research studies as conventionally conceived and 
designed? The technical work under this and the successor grant will 
converge with the philosophical argument in thr 1974 paper, to raise 
sharp questions about the appropriate function of social-science inquiries 
pointed toward social policy. The contribution emerging from the grant 
is the evidence that studies of reasonable size probably cannot pin 
down quantitative effects definitively. 



0 

In 1940 Austin J'.ond pubHr.h-?d n doctoral dissei*tiU' ion of unuaunl 
uethoJoXoniccil quality, much innutmccd by Ik^len W.ilker. lie taujjht 
ftcncticsi in college by two procedures, one traditional mid the other 
with conctaui aLtcution to social itnpl lent ions?. *'slng tlie Johnfson- 
Neymau method of atatiatical aivilysis, ha rcpoi . - significant differences 
an Many out comes » but showed that the effects shifted frora strong to 
ne^tllgiblc or even to reverse effects* depending on the characteristics 
oC the studonl al the Ktart of the course. This data set was used for 
our applicciLion of Potthoff's extension of the John son-Key man method. 
(The study, as reported, does not lend ItBelf to a separation of class 
and individual effects as only two clnsp.es were involved.) 

Fiiiurc 1 displays confidc-nce limits for one of Bond's outcomes. 
Ills analysis established rcydoas of niKnifScanre. These regions appear in our 



Insert Fixture 1 here 



figure in the horizontal plane tlirough the origin, bounded by an hyperbola. Only 
for persons in the outer corners could Bond conclude that the treatments 

differed significantly. The three-dimensional display given by Potthoff s 

method augments this information with an estimate of the possible range 

of the treatment effect at each point in the space defined by the 

pretests. Also, the curved surface defines the limiting positions of 

the regression plane. Any plane that docs not slice into the surface 

might be the regression plane that describes the treatment difference in 

the population. This is a pow:frfuJ method of taking into account the 

uncertainties introduced by quasicolilnoarity , a problem of considerable 

ERIC 
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Figure 1, 



Confiuonce liinits for an interaction effects in Bond'f* 
Bond's Ftudy of genetics. EA is a u«st on Evaluation 
of Authorities. 
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importance in iuterprt't itip, inultlple rogrosslons. 

Our tiiathcmatjcal j.iodol c;ni only be. skctchcU here. First we 
hypcthcsiiu that, vithl;^ n treatment, tho cut come Y depends on three 
UjndK of f.;ffccC. 

1. Indlvluunl effect of nptJtude }[. Equals the expected 
Y for {»ersoa^^ with a pnrti.culnr levtil of X^i averaged 
over uU the cln«st;s in tho populntJon, 

2. Class effect. Eqiinl« th4» Y value for the class, after 
adjusting +oi the individual effect of the members. 
incJudt's n»iy cnpericnces corinon to the ^roup (e.g., 
teacher o-.ctV "' ncti, or hif.h {.;roup moral e). This effect 
TPay be a function c*" the cla:iK mean on X. 

3. Indlvidual-\.;i.thin-clar.s effect. Described by the regression 
of Y or >: within the class, adjusted for Effect 1. Reflects 
any bpcci.-;! di.^trlbution of opportunity, cncouraper.icnt , 

or' the .like v;iihia lite claK.s (e.g., teacher slows pace 
to fit the weaker students). 

Now each of these effects may ditfcr frora one treatsacnt to another. 

lUuicc there arc three kindu of "Aptitude y- Treatment interaction". 

having diffr-rtnt theoretical and practical implications. Tliis has not 

previously been recogniJied. 

Analyzing data within a treatment, one obtains a between-classe-s 
slope and a within-class slope for the class. ITius one has three "unknowns" 
(three effect sises) and only two observations (slopes). One can regard 
the between slope as a composite of Effects 1 and 2, the within slope 
as a composite of 2 and 3. 

The sampling model we emphaslEe at present regards the set of 
classes as randomly sampled frc > whe population of classes receiving 
the samo treatment. This allows us to consider quasiexpcriments in 
which nonrandoM causes determine- whet treatment a particular class 
gets. (Even true experimonts, if conducted on a large scale, are 

o 
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Ifkiily to bo rcUucoO lo quas i cxpc -Im'taM when hovh\ cl.nr.BOii Uepart from 
the plan.) Wc tn^a* puj)!!?? fixed within clai;r»cr.* In future work 

Bhall. trace tin* porf^iblli ticn, 1 Ind ratic»rs^ and procochircj^ ju^soclntcnt 
vith alurrualive lUv^dcla. 

Wc have to date operaUcU as if the pupil X score is fixed. The 

work need'j to be extended to eonaider measurement error (regarding X an 
a random observation from a eoLlection of 5i obF.ervutious on the fixed 
pupil). The rej^ulting "correction for attenuation** takes a different 
foim in studies with nested data than it does in he usual individual 
Btudy. 

Inner jnj; irxasurejnent error and analy?5inj» solely within a treatment, 

we consider tlie out cone score Y of person p in class c to be ntade up of 

pc 

co:.ip>n('!^ts: 

(1) Y » $ General mean 

pc o 

^• (>/x Betwccn-class re^',ression 
X c 

+ B C (nar>H-lcvcl residual; C is a dunnny variable, 
c 

+ P X Expected (over cla&sen) witbln-class 

^ rei;ression. X,, -X^ 

A. (ct (> sv: 2p P c 

^^2c 2 2p Spacific within-class regression 

*f 6 Residual 

The first three terms add to Y . Hence we make a decomposition: 

c 

(2) SS(Yp^) = SS(Y^) + SS(Yp^-y 

Total Between Within 

classes rlanscs 

Any paraniater which is a function of the clj;f.s means (v^, or o^(y), 

or , etc.) can be defined taking class size into account as a weight » or 

walr,htin>i clai;r,c;« i*nunl3y. The value of one or uoYC parnnetcrB v;lll rJiift betv/een 
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Figure 2, Anderson presentation of an interaction findings 
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iho Lwo calrul at iiulcss Lhc Y dist ribuliorx is the sarna for lar^.e 
classes* UV h;nc i^lioson to est i;aaLe parameters in the weighted 
• wiiii^lt c<;^v»il>. IhLs has not buon the usual prac- 

; ice in oJucalici^il research. The argiiwouts for and against this 
fojnulaLion /ire to appoar in our technical report. 

The betv^eeu-ol:ij;s sum r^quares is now decomj^osed 
{ I) SS(V > SS.Ucfxr + SS- Residual 

C i X 

Aptitude Clacs 
effv.ct effect 

ilicst i\o suiub; of >5quareG cor.c cmt at the first step of a generalized 

rt^^rcsoiori nualyr^is with (1) at^ the models For that analysis, one must 

icu;> every stud^^Tit (not eveiy class) ar» n case, 

•j'ite wUhin-c lc>ss docoiKi^o.sit ion rcc;uLres tv;o more steps of regression. 

(i) ?S(Y ,"7 ) - SS Ucgr + (SSJ^egr-SS Rcftr) + SSJtesidual 

pC C ^ mf ^ »^ 

Ccnoral within- Specific within- Unpredicted 

claf>s aptitude class aptitude 

effect effect 

Thv strenglh of various effects is now estimated by the proportion 
of r»S(Y) in each of the five se^^cnts. 

The model jiuneraiiscs to more than one aptitude* We find that for 
results to be Interpreted it is uecesary to convert the aptitude to 
dimensions that are close to orthogonal (in the pooled-^within-treatments 
distribution) • 



The. ncixt question Imxh to do vith infcvciices about the size of 
effects in t\w populaticm. Uncltr the Hanii)ling modal emphasized above, 
one can look nt confidence intervals on the effects at the class level 
(the general mean aiid the between-class slope), following the Potthoff 
rationale. Almost invariably, the confidence intervals will be quite 
wide becaur.e data are i^vailable on only a few classes. Hence it x^ill 
rarely he possible to draw definite conclusions about population para- 
meters in instructional cxpcrimentB of a practical size. As we see it, 
the use of the number of pupils as the basis for calculating confidence 
Intervals or significance tests requires a special justification that 
may not often apply. We will spell out the grounds for such justifica- 
tion in next year's work. 

I£ one is prepared to regard assignment of classes to treatments 
as random, the generalized regression analysis can be used to test the 
significance of aptitude-treatment interactions or the Potthoff method 
can be applied to the interaction. 

The sketch above speaks of a single predictor X which is 
decomposed into between and within-class components • All the statistical 
arguments developed apply to multivariate prediction, as in the Anderson 
study* 

The Anderson study reanalyzed 

In a Minnesota doctoral dissertation completed in 1941, G. L. 
Anderson compared drill methods and meaningful methods of instruction in 
17 fourth-grade class classrooms. The Johnson-Neyman analysis indicated 

differences on several outcomes, and showed that the effects depended on 
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Table 1 

Regression Coefficients predicting Overall Outcome 
in the Anderson Data 



2 2 

Treatment Multipl e regr er,r> ion s (Y-X) Simple reg ression s (Y-X) 

\bil ^Precoii. ^Abil ^Precom Abil Frecom 

Between classes D 0.06 0.75 499 -^.47 0.74 2063 498 



M 



0 43 0.70 705 0.22 0.47 1365 1153 



Within cUiases D 0.41 0.75 2679 0.39 0.73 6656 3980 

M 0.42 0.65 2825 0.51 0.71 6242 4369 
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the abilities of the studnnt. at the bc'i'.i.nning of the year. Anderson 
divult'd the cJa.s.-:fs inlo two HubRroiJps (liir.h or low moan initial abilities), 
and laado separate analyses iu the subgroups, counting the ^dividual student 
a« Uie naupllug unit, ilis results arc typified by Figure fvora his short 
publi-.au-d report. It: looked as if (under the conditions of the experi- 
Ttjcnt) the uieantugful methods then beln^ advocated by educational theorists 
workc.d tor underachie^'ors aud were haimtui — relative to drill — for 
overncliiovcrs. This vas a kind of answer to Binct's old problem of doing 
soMCtiiinr, about tlie "brieht" pupil with a poor school record. It has its 
nodoni echo in various statcn'.enLs about the wethod? that work best with 
middle-class and lower-class children. 

Aiidcrson did not consider class effects in hjs regression analyses. 
We analyzed the data to separate class and individual effects. 

In preparation for the reanalys.ls we dropped some cases with 
iai«sins data (reducing N to /!35) , made regression cHtirantes of missing 
scores for others, and eliminated one small class. We did not form sub- 
groups with different initial abilities as Anderson had done. For the 
analysis reported here we fornjed a composite dependent variable by com- 
bining the Compass and van Wagenen post tests, Anderson used the several 
subtests as dependent variables. As predictors we used two aptitude 
variables: the Compass pretest and the Minnesota School Ability test. We 
converted the latter to ABIL (=MSAT - 0.42 Compass) to get orthogonal 
predictors. (Within a class or treatment the correlation was not neces- 
sarily zero.) We rescaled the two predictors and dependent variable to 
mean zero and s.d. 1.00, over all cases pooled. 
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Thu hii^As for de.[;rues of freedom was not the number of students, but 
the number oi classes. The small number of degrees of freedom made the 
confidence intervals for all betvcecn-class regression lines or planes 
very wide. 

Our origiital analysis was a multiple roi-ression. The multiple 
rei;ression coefficients (Table 1) between classes are consistent with 
AnJei-Bon's finding, since the overachiever is a person high on ABIL, 
and overachieving clasney see© to do better in the Meaning treatraent. 

The computer printout drew our attention to the fact that ABIL accounts 
for very little variance in the Dri3,l group. This led us to examine the 
plot of cla.ss meana for raw scores (Figure 3). It is evident that Anderson's 
treatment groups had markedly dlssimilnr distributions of class means. He 
had assigned classes to methods on the basis cf the teachers' preferences, 
and happened — in a small sample -~ to draw a set of Drill classes in 
which MSAT and PRECOM weann correlated highly. The collinearity in the 
Drill classes nearly wiped out the variance of ABIL, the partial variate. 
Consequently, the multiple-regression weights in the between-classes 
analysis for Drill are not interpretable; many other sets of weights 
would give nearly the same multiple correlation. 



Insert Figure 3 here 

While we calculated within-class multiple regressions for each 
class, it is more useful here to report the calculation for classes pooled. 
The within-class estimates represent &Z' There is obviously no difference 
between the wLthin-clasG rej?,reyr,ion slopes. 
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Figure 4. Plot of class means on 

post test against pretests. 



To rtiaeh couclu:;ions lesa pcrt:urbcd l>y colllnearity at the 
botWi'ua--c1 a^is^'ii lovd \:c c:\^ LulcXcd fjinplc roni f'r>sion slopes. Again^ the 
wJthin-group slopes diflcr iittlo. The difference betww^uu 0.39 and 0.51 
for A15IL is in line with Andersen's couciuBXon but is a very weak clfcct. 

Tho differences in the betwecn-groups analysiii are impressive^ 
and entirely consistent with Anderson's conclusion, lluwever, one must 
rGcojini?:e the very large sampling error in a slope based on 8 or 9 
classes. A plot of the means for AFIL, PllECOH, and ZACH (the standardized 
outcome) appears in Figure 4. 



Insert Figure 4 here 



In the chart lor ABIL, the narrow range of ABIL in the Drill group 
should be noted. This lollows from the collinearity . The slope is 
determined quite unreliably. In fact, if it were not fcr the one class at 
-55, fi.%thc slope in the Drill group woulo be slight.l y positive not 
negative. In th- *eaning group, the slope is low, and not reliably positive. 
The two set. of points could easily be from the same distribution, hence 
we cannot say that the data support an ATI hypothesis. Yet this between- 
group effect must be the inajor source of Anderson's interaction* 

In the PRFCOM chart it is even more obvious that the two sets of 
cases fonu one distribution. The two leftmost cases account for the 
steeper slope in the Drill treatment. 

Finally, it is noted that the residual variances are not so different 
as to suggest an important difference between treatments. 

The variance decomponition (as far as we have carried it) is a.> 
follows! 
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Figure 3. Plot of class means on pretests 
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Total SS 


9920 


100.0% 


1000 


100.0% 


Between classes 


2134 


21.5% 


1498 


14.9% 


Predicted 


1645 


16.6% 


793 


7.9% 


Residual 


A99 


4.9% 


705 


7.0% 


Within classes 


7788 


78.5% 


8560 


85.1% 


A 

Predicted by 02 


5109 


51.5% 


5735 


57.0% 


Residual 

1 


2679 


27.0% 


2825 


28.1% 



It remains to comment on the specific within-class analyses. With 
about 20 cases in most classes, regression slopes are not to be regarded 
as precise, even when pupils are considered to be fixed. The slopes 
onto PRKCOM ranged from 0.51 to 1.07 over the DRILL classes, and from 
0.43 to 1.12 over the MEAIUNG clasi?es. The largest differences may imply 
that ihe teachers at the extremes were following quite different practices. 
As for ABIL, the range of within-class slopes was 0.14 to 0.92, and 0.22 
to 0.71, in the respective treatments. 
Conclusion 

All in all, we are satisfied that our approach adds a great deal to 
the understanding of educational data and is likely to alter conclusions 
in many studies. 

There are formal statistical problems to think through and explicate. 
Our computational methods can be made more efficient. And as 
we apply the method to further bodies of data in the present year we will 
no doubt learn more about the limits of the method and its value. 
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